#+title: neural-turing-machine

#+date: <2018-05-09>

#+begin_quote
  One way RNNs are currently being used is to connect neural networks
  more closely to traditional ways of thinking about algorithms, ways of
  thinking based on concepts such as Turing machines and (conventional)
  programming languages. [[https://arxiv.org/abs/1410.4615][A 2014
  paper]] developed an RNN which could take as input a
  character-by-character description of a (very, very simple!) Python
  program, and use that description to predict the output. Informally,
  the network is learning to "understand" certain Python programs.
  [[https://arxiv.org/abs/1410.5401][A second paper, also from 2014]],
  used RNNs as a starting point to develop what they called a neural
  Turing machine (NTM). This is a universal computer whose entire
  structure can be trained using gradient descent. They trained their
  NTM to infer algorithms for several simple problems, such as sorting
  and copying.

  As it stands, these are extremely simple toy models. Learning to
  execute the Python program =print(398345+42598)= doesn't make a
  network into a full-fledged Python interpreter! It's not clear how
  much further it will be possible to push the ideas. Still, the results
  are intriguing. Historically, neural networks have done well at
  pattern recognition problems where conventional algorithmic approaches
  have trouble. Vice versa, conventional algorithmic approaches are good
  at solving problems that neural nets aren't so good at. No-one today
  implements a web server or a database program using a neural network!
  It'd be great to develop unified models that integrate the strengths
  of both neural networks and more traditional approaches to algorithms.
  RNNs and ideas inspired by RNNs may help us do that.
#+end_quote

Michael Nielsen,
[[http://neuralnetworksanddeeplearning.com/chap6.html#other_approaches_to_deep_neural_nets][Neural
networks and deep learning]]
